Papers with linear classifier
LLMs Know More About Numbers than They Can Say (2026.eacl-short)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly used in mathematical, scientific, financial and engineering domains. |
| Approach: | They probe the hidden states of several smaller open-source LLMs to find out how big they are . |
| Outcome: | The proposed model improves verbalized accuracy by 3.22% over base models. |
Beyond Fine-tuning: Few-Sample Sentence Embedding Transfer (2020.aacl-main)
Copied to clipboard
| Challenge: | Fine-tuning (FT) pre-trained sentence embedding models on small datasets has been shown to have limitations. |
| Approach: | They propose to combine embeddings from a pre-trained model with a simple sentence embeddable model. |
| Outcome: | The proposed approach outperforms FT on small datasets with negligible computational overhead. |
Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps (2024.emnlp-main)
Copied to clipboard
| Challenge: | Despite the utility and impressive capabilities of large language models, their tendency to generate hallucinations presents a significant challenge in their deployment. |
| Approach: | They propose a simple hallucination detection model based on the ratio of attention weights on the context versus newly generated tokens. |
| Outcome: | The proposed model reduces the amount of hallucinations by 9.6% in a summarization task. |
Improving Informally Romanized Language Identification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Latin script is often used to informally write languages with non-Latin native scripts, resulting in high spelling variability. |
| Approach: | They propose to improve methods used to synthesize training sets to incorporate natural spelling variations into training sets. |
| Outcome: | The proposed method improves test F1 from the reported 74.7% (using a pretrained neural model) to 85.4% (using the linear classifier trained solely on synthetic data). |
Safe-Unsafe Concept Separation Emerges from a Single Direction in Language Models Activation Space (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing approaches to ensuring the safety of Large Language Models (LLMs) rely on invasive fine- tuning or external generation-based checks, which can be opaque and resource-inefficient. |
| Approach: | They propose a mechanistic method that identifies the layer where safe and unsafe concepts are maximally separable within a pretrained representation space. |
| Outcome: | The proposed method can be used across multiple domains, diverse tasks, and 16 non-English languages on encoder and decoder architectures. |
Improving Commonsense Question Answering by Graph-based Iterative Retrieval over Multiple Knowledge Sources (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods to facilitate natural language understanding rarely involve commonsense or background knowledge. |
| Approach: | They propose a question-answering method that integrates multiple knowledge sources to boost performance. |
| Outcome: | The proposed method outperforms other competing methods on the CommonsenseQA dataset and achieves the new state-of-the-art. |
It’s All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning (2021.findings-acl)
Copied to clipboard
| Challenge: | gilbert et al.: commonsense reasoning is a key problem in natural language processing but its capabilities are still unstudied. gilland eetal.: a new approach to commonsensible reasoning is needed to solve the problem. |
| Approach: | They propose a method which trains a linear classifier with weights of multi-head attention as features and a multilingual Winograd Schema corpus to measure cross-lingual generalization ability. |
| Outcome: | The proposed approach performs competitively with recent approaches even when applied to other languages in a zero-shot manner. |
GBT: Generative Boosting Training Approach for Paraphrase Identification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Paraphrase Identification (PI) is a fundamental natural language understanding task with non-trivial challenges. |
| Approach: | They propose a Generative Boosting Training approach for Paraphrase Identification (PI) they use a seq2seq model to perform DA on misclassified instances periodically . |
| Outcome: | The proposed method outperforms state-of-the-art PI models on English and Chinese PI tasks with good efficiency and effectiveness. |
CHiLL: Zero-shot Custom Interpretable Feature Extraction from Clinical Notes with Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study suggests that linear models with interpretable features are more reliable than opaque models. |
| Approach: | They propose an approach for natural-language specification of features for linear models . they prompt LLMs with expert-crafted queries to generate interpretable features from health records . |
| Outcome: | The proposed approach can be used to craft features clinically meaningful for downstream tasks . it is based on a risk prediction task and standard predictive tasks based upon this data . |
ParlVote: A Corpus for Sentiment Analysis of Political Debates (2020.lrec-1)
Copied to clipboard
| Challenge: | Debate transcripts from the UK Parliament contain information about the positions taken by politicians towards important topics, but are difficult for humans to process manually. |
| Approach: | They propose to use a linear classifier and a transformer word embedding model to classify sentiment polarity in debate speeches to evaluate sentiment analysis systems for the political domain. |
| Outcome: | The proposed method performs better on the largest dataset and is more robust to other datasets. |
Sparse Logistic Regression with High-order Features for Automatic Grammar Rule Extraction from Treebanks (2024.lrec-main)
Copied to clipboard
| Challenge: | Descriptive grammars are valuable, but they lack quantitative data. |
| Approach: | They propose to extract and explore significant fine-grained grammar patterns and potential syntactic grammar rules from treebanks to create an easy-to-understand corpus-based grammar. |
| Outcome: | The proposed model captures well-known and less well- known significant grammar rules in Spanish, French, and Wolof. |
Word Salad Chopper: Reasoning Models Waste A Ton Of Decoding Budget On Useless Repetitions, Self-Knowingly (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Reasoning Models (LRMs) are often bottlenecked by the high cost of output tokens. |
| Approach: | They propose a lightweight, turnkey component for Large Reasoning Models that is minimally invasive to its reasoning trajectory. |
| Outcome: | The proposed component is lightweight and low overhead, and lacks semantic value. |